feat: #1831 Add opt-in cost tracking for LiteLLM models #1832
Conversation
seratch
changed the title
|
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
| """Whether to include usage chunk. | ||
| Only available for Chat Completions API.""" | ||
|
|
||
| track_cost: bool | None = None |
There was a problem hiding this comment.
This is only related to LiteLLM use cases. Even if we decided to have this feature, I don't think this property should be here. If we add something like this, we may want to introduce LiteLLMSettings, which has only track_cost so far, and pass it to LiteLLMModel constructor as a new arg.
There was a problem hiding this comment.
I actually like this idea. I thought it could be too big of a change, but if this is welcome I will go ahead with this.
| ): | ||
| if isinstance(event, ResponseCompletedEvent): | ||
| # Extract cost if it was attached by LiteLLM model. | ||
| cost = getattr(event.response, "_litellm_cost", None) |
There was a problem hiding this comment.
Sorry, but I still hesitate to rely on LiteLLM's underscored property
There was a problem hiding this comment.
How about storing the cost in a _last_stream_cost attribute on the LitellmModel instance instead, and have run.py extract it from there?
Resolves #1831
Adds cost tracking for LiteLLM models when users enable it via
ModelSettings(track_cost=True).Changes
costfield toUsageclasstrack_costsetting toModelSettings. Opt-in (defaults toFalse)_hidden_params["response_cost"]litellm.completion_cost()